Coverage integration test for requirements #152

TannazVhdBMWExt · 2024-12-10T15:19:44Z

No description provided.

four different coverage tests have been added

...alid-source-input-file-paths/invalid-directory-input-scenario/expected-output/output.lobster

tests-integration/projects/coverage-half/Makefile

phiwuu · 2025-01-29T11:28:25Z

tests-integration/projects/coverage-mix/Makefile

see comments in above Makefile

phiwuu · 2025-01-29T11:28:32Z

tests-integration/projects/coverage-zero/Makefile

see comments in above Makefile

phiwuu · 2025-01-29T11:28:38Z

tests-integration/projects/coverage/Makefile

see comments in above Makefile

phiwuu · 2025-02-04T12:13:55Z

tests-integration/projects/coverage-half/Makefile

+THIS_TEST_ESCAPED:=$(subst /,\\/,$(THIS_TEST))
+REFERENCE_OUTPUT:=report.reference_output
+
+html_report.html: softreq.lobster sysreq.lobster lobster.conf trlc-softreq.conf trlc-sysreq.conf


this make target calls lobster-report, and hence no file called html_report.html will ever be generated.
Did you mean lobster.report instead?

phiwuu · 2025-02-04T12:15:57Z

tests-integration/projects/coverage-half/Makefile

+	@if diff report.lobster $(REFERENCE_OUTPUT); then \
+	    echo "Files are identical"; \
+    else \
+        echo "Files are different"; \
+        exit 1; \
+	fi


please fix the indentation

phiwuu · 2025-02-04T12:18:03Z

tests-integration/projects/coverage-mix/Makefile

+html_report.html: softreq.lobster sysreq.lobster testreq.lobster lobster.conf trlc-softreq.conf trlc-sysreq.conf trlc-testreq.conf
+	@lobster-report
+	@if diff report.lobster $(REFERENCE_OUTPUT); then \
+	    echo "Files are identical"; \
+    else \
+        echo "Files are different"; \
+        exit 1; \
+	fi


same as above:

no file called html_report.html will ever be generated; you probably meant report.lobster

please fix the indentation

phiwuu · 2025-02-04T12:21:54Z

tests-integration/projects/coverage-mix/example.rsl

+  text String
+  derived_from optional Software_Requirement  [1 .. *]
+  just_up   optional String
+  just_down optional String


Why did you introduce just_down? The tracing policy in lobster.conf shows that "Test Requirements" are on the lowest level. There is no level beneath them. just_down can never have any impact in this test setup. Or did you want to verify this fact?

phiwuu · 2025-02-04T12:24:08Z

tests-integration/projects/coverage-mix/testreq_example.trlc

+   text = '''Requirement3'''
+   derived_from = [softreq_coverage_half_example.requirement_d]
+   just_up = "needed"
+   just_down = "needed"


just_down will not have any impact in this test. It can be removed.

phiwuu · 2025-02-04T12:27:47Z

tests-integration/projects/coverage-zero/Makefile

+html_report.html: softreq.lobster sysreq.lobster lobster.conf trlc-softreq.conf trlc-sysreq.conf
+	@lobster-report
+	@if diff report.lobster $(REFERENCE_OUTPUT); then \
+	    echo "Files are identical"; \
+    else \
+        echo "Files are different"; \
+        exit 1; \
+	fi


same as above:

no file called html_report.html will ever be generated; you probably meant report.lobster

please fix the indentation

phiwuu · 2025-02-04T12:28:57Z

tests-integration/projects/coverage/Makefile

+html_report.html: softreq.lobster sysreq.lobster lobster.conf trlc-softreq.conf trlc-sysreq.conf
+	@lobster-report
+	@if diff report.lobster $(REFERENCE_OUTPUT); then \
+	    echo "Files are identical"; \
+    else \
+        echo "Files are different"; \
+        exit 1; \
+	fi


same as above:

no file called html_report.html will ever be generated; you probably meant report.lobster

please fix the indentation

kedarnn · 2025-02-05T06:01:04Z

tests-integration/projects/coverage-half/Makefile

+	@if diff report.lobster $(REFERENCE_OUTPUT); then \
+	    echo "Files are identical"; \
+    else \
+        echo "Files are different"; \
+        exit 1; \
+	fi


Suggested change

@if diff report.lobster $(REFERENCE_OUTPUT); then \

echo "Files are identical"; \

else \

echo "Files are different"; \

exit 1; \

fi

@if diff report.lobster $(REFERENCE_OUTPUT); then echo '✅ Files are identical!'; else { echo '❌ ERROR: Files are different!'; exit 1; }; fi

kedarnn · 2025-02-05T06:04:20Z

tests-integration/projects/coverage-mix/Makefile

+	@if diff report.lobster $(REFERENCE_OUTPUT); then \
+	    echo "Files are identical"; \
+    else \
+        echo "Files are different"; \
+        exit 1; \
+	fi


Same as https://github.com/bmw-software-engineering/lobster/pull/152/files#r1942289849

kedarnn · 2025-02-05T06:04:42Z

tests-integration/projects/coverage-zero/Makefile

+	@if diff report.lobster $(REFERENCE_OUTPUT); then \
+	    echo "Files are identical"; \
+    else \
+        echo "Files are different"; \
+        exit 1; \
+	fi


Same as https://github.com/bmw-software-engineering/lobster/pull/152/files#r1942289849

kedarnn · 2025-02-05T06:05:01Z

tests-integration/projects/coverage/Makefile

+	@if diff report.lobster $(REFERENCE_OUTPUT); then \
+	    echo "Files are identical"; \
+    else \
+        echo "Files are different"; \
+        exit 1; \
+	fi


https://github.com/bmw-software-engineering/lobster/pull/152/files#r1942289849

mugdhadhole1 · 2025-02-05T07:05:15Z

tests-integration/projects/coverage/example.rsl

@@ -0,0 +1,12 @@
+package coverage_example


Package names in the rsl files are same and it shows the following error message

duplicate definition, previous definition at tests-integration/projects/coverage-half/example.rsl:1

Just for clarification: it is okay to have duplicates across different test cases. But if you use the TRLC extension for VSCode, and if it scans the whole repository, then it will complain about duplicated definitions.

mugdhadhole1 · 2025-02-05T07:06:27Z

tests-integration/projects/coverage-zero/example.rsl

@@ -0,0 +1,12 @@
+package coverage_example


same as https://github.com/bmw-software-engineering/lobster/pull/152/files#r1942343472

phiwuu · 2025-02-05T09:17:28Z

tests-integration/projects/coverage-mix/lobster.conf

+requirements "Software Requirements"
+{
+   source: "softreq.lobster";
+   requires: "Test Requirements";


Why did you add the requires keyword here? Since "Test Requirements" already use trace to: "Software Requirements"; it will not impact the tool behavior. Or was the idea to test exactly that the statement has no impact?

After investigation it seems that lobster-report behaves like follows (but should not):

if requires is specified for a level (here: "Software Requirements"), then any implicit conditions imposed through trace to coming from other levels are ignored.

That's why the coverage is 100%, where it should really be 0%.

Recommendation:
The integration tests shall focus on the trace to feature only. Please remove requires from the tests. The behavior of requires will be replaced by a new feature called trace from with pull request #91 anyway.

phiwuu · 2025-02-05T09:48:59Z

tests-integration/projects/coverage-mix/testreq_example.trlc

+{
+   text = '''Requirement3'''
+   derived_from = [softreq_coverage_half_example.requirement_d]
+   just_up = "needed"


Why do you give a justification in this test, while adding a link to requirement_d at the same time?

phiwuu · 2025-02-05T10:25:23Z

tests-integration/projects/coverage-mix/report.reference_output

+          "status": null
+        }
+      ],
+      "coverage": 100.0


The requirement_d is not linked to any requirement from level "System Requirements", but the tracing policy enforces that through trace to. As stated above, the requires keyword apparently removes this strict condition, which is a bug. Please do not use requires in the tests, and set the expected coverage to 0 here.

phiwuu · 2025-02-07T12:01:40Z

tests-integration/projects/coverage-mix/example.rsl

@@ -0,0 +1,22 @@
+package coverage_half_example


The test case is called coverage-mix, not coverage_half_example. It is very confusing to give the TRLC package the name of a different test case.

phiwuu · 2025-02-07T12:04:02Z

tests-integration/projects/coverage-mix/lobster.conf

+
+requirements "System Requirements" {
+   source: "sysreq.lobster";
+   trace to: "Software Requirements";


It is very confusing to derive system requirements from software requirements. Please consider to either switch the names, or use made-up names like "Apple", "Banana", etc. instead!

phiwuu · 2025-02-07T12:05:47Z

tests-integration/projects/coverage-mix/example.rsl

+type Software_Requirement
+{
+  text String
+  derived_from optional System_Requirement  [1 .. *]


The one requirement in softreq_example.trlc does not use the attribute derived_from at all. It can be removed.
Furthermore, the example.rsl demands that the breakdown happens the other way around, from software requirements to system requirements.
Please keep the test as simple as possible, and don't pollute it with configuration aspects that are not important to the test.

TannazVhdBMWExt requested a review from a team as a code owner December 10, 2024 15:19

phiwuu added the internal Affects the CI, tests or refactorings only, not relevant to the end-user label Dec 10, 2024

TannazVhdBMWExt and others added 3 commits December 11, 2024 15:40

Tests adaptation

0d854ef

Merge branch 'bmw-software-engineering:main' into main

5dfff31

Coverage integration tests

fc04145

four different coverage tests have been added

TannazVhdBMWExt force-pushed the coverage-integration-test branch from 600c9e5 to fc04145 Compare December 12, 2024 14:26

TannazVhdBMWExt and others added 2 commits December 12, 2024 09:36

Merge branch 'bmw-software-engineering:main' into main

3d01128

Merge branch 'main' into coverage-integration-test

8a67af6

TannazVhdBMWExt changed the title ~~Coverage integration test for two requirements~~ Coverage integration test for requirements Dec 12, 2024

TannazVhdBMWExt force-pushed the coverage-integration-test branch from 156fa3c to 46c3030 Compare December 12, 2024 14:57

Coverage integration tests

a7ac888

four different coverage tests have been added

TannazVhdBMWExt force-pushed the coverage-integration-test branch from 9483724 to a7ac888 Compare December 13, 2024 14:10

TannazVhdBMWExt and others added 4 commits December 13, 2024 16:21

Correct the tests reference files

47ceec5

Merge branch 'bmw-software-engineering:main' into main

b243242

Merge branch 'main' into coverage-integration-test

a30880a

revise

704aba9

phiwuu requested changes Jan 29, 2025

View reviewed changes

...alid-source-input-file-paths/invalid-directory-input-scenario/expected-output/output.lobster Outdated Show resolved Hide resolved

phiwuu requested changes Jan 29, 2025

View reviewed changes

apply PR review

8dafc12

phiwuu requested changes Feb 4, 2025

View reviewed changes

kedarnn reviewed Feb 5, 2025

View reviewed changes

mugdhadhole1 reviewed Feb 5, 2025

View reviewed changes

phiwuu requested changes Feb 5, 2025

View reviewed changes

phiwuu requested changes Feb 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Coverage integration test for requirements #152

Coverage integration test for requirements #152

TannazVhdBMWExt commented Dec 10, 2024

phiwuu Jan 29, 2025

phiwuu Jan 29, 2025

phiwuu Jan 29, 2025

phiwuu Feb 4, 2025

phiwuu Feb 4, 2025

phiwuu Feb 4, 2025

phiwuu Feb 4, 2025

phiwuu Feb 4, 2025

phiwuu Feb 4, 2025

phiwuu Feb 4, 2025

kedarnn Feb 5, 2025

kedarnn Feb 5, 2025

kedarnn Feb 5, 2025

kedarnn Feb 5, 2025

mugdhadhole1 Feb 5, 2025

phiwuu Feb 7, 2025

mugdhadhole1 Feb 5, 2025

phiwuu Feb 5, 2025 •

edited

Loading

phiwuu Feb 5, 2025

phiwuu Feb 5, 2025

phiwuu Feb 7, 2025

phiwuu Feb 7, 2025

phiwuu Feb 7, 2025

Coverage integration test for requirements #152

Are you sure you want to change the base?

Coverage integration test for requirements #152

Conversation

TannazVhdBMWExt commented Dec 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phiwuu Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phiwuu Feb 5, 2025 •

edited

Loading